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ABSTRACT 



Relatively diminishing financial resources and increasing 
user expectations are combining to create a new emphasis for the traditional 
research library. This paper reports on the development of a pilot project at 
a large research library that explored the relationship between evaluation 
and assessment to determine how both the institution and the user can 
benefit. Following a background of needs of the University of Illinois at 
Urbana -Champaign Library, the survey methodology for determining user needs 
is outlined. The survey was divided into three distinct areas. In the first 
part of the survey, users were asked to identify their relationship with the 
library. The second part contained assessment questions which attempted to 
ascertain users* information literacy and general search skills. Finally, 
questions specific to the University's system were asked. The results of the 
Task Force on the Future of the Library and a changed university budgetary 
process brought about the recommendation to develop/fund a Instruction 
Coordinator position and include support staff with a real budget for the 
implementation of a library-wide program. (AEF) 
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Relatively diminishing financial resources and increasing user expectations are combining to create a new 
emphasis for the traditional research library. Although collection development continues as a major factor 
in the life of such libraries, user services are being increasing emphasized as the academic world 
incorporates more and more in the way of management techniques from the business world. Many libraries 
have been affected by the changing business management styles that have come into vogue in the last 
twenty years. This paper reports on the development of a pilot project engaged in the combination of 
evaluation and cognitive and skills assessment of users to provide for effective strategic planning for 
library instruction at a large research library. 

Assessment and Evaluation - An Attempt at Deflnition 

Whether the changes from the traditional bureaucratic model of dependence upon a professional elite to 
extreme responsiveness to user needs is beneficial for the institution is the topic for another study. Here, 
we will explore the relationship between evaluation and assessment to determine how both the institution 
and the user can benefit. Evaluation and assessment work together to provide a basis for planning, but in 
many ways they are at opposite ends of the user spectrum. In this instance, the word evaluation will be 
directed toward a normative process. Evaluation is a judgment call. When evaluation is done in the library, 
it is most commonly done on the basis of one individual's opinion and as the word stem would indicate, 
values play a large part in the process. Assessment on the other hand implies a more empirical approach to 
a question. This is not to say that assessment is value free. Indeed, if we are to believe our postmodern 
philosophers, even the most empirical test is not free from some type of normative process. The results of 
studies can certainly be affected by the manner in which results are read. The manner by which the test was 
created, the sample taken, and the means used to compile information can also play a large part in the 
normative structuring of the empirical study. For our purposes, however, the implication is that general 
agreement has been reached by several people or institutions concerning what will be assessed and how it 
will be assessed. At least enough consensus has been achieved to validate the assessment on the part of a 
majority of those who might form the knowledgeable elite. 

Recent History - The Growth of the Importance of Assessment 

The increasing importance of assessment is underlined in higher education by the way in which regional 
accrediting agencies in the United States have been approaching the reaccreditation process. There are six 
agencies in the country that routinely visit colleges and universities in their geographic region and provide 
an avenue for credentialing the institutions as meeting certain standards of operation. The accreditation 
process is purely voluntary, but very important to the well-being of most institutions of higher learning in 
the United States. Through agencies such as the North Central Association of Colleges and Schools, higher 
education seeks to set standards of quality while at all times seeking improvement in the programs 



provided to students. In the past, these accrediting agencies were primarily concerned with evidence in the 
areas of governance and administration, finance, admissions standards and the provision of student 
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services, institutional resources such as the number of books and types of services provided by libraries, 
student academic achievement, institutional effectiveness, and the relationships between the institution and 
its outside constituencies. The college or university was charged with writing a self-study, covering certain 
specific factors that the agency determined as essential to a good program. Within the past decade, the 
emphasis of the North Central Association has changed from focusing on institutional assets to focusing on 
outcomes assessment. To see an overview of the criteria go to 

http://www.ncacihe.org/overview/ovcriteria.cfm. The result of this change in emphasis by the accrediting 
agencies has been change on the part of many administrations in the way budgets are meted out to the 
distinct parts of institutional operations. Assessment and evaluation within individual departments or areas 
of operations are becoming more commonplace as a result of this movement. 

A Case in Point 

The changing emphasis mentioned above, plus the migration to a new catalog interface for the University 
greatly increased the awareness of user needs, attitudes, and abilities for many of the librarians working at 
the University of Illinois at Urbana-Champaign (UIUC). The University supports the largest publicly 
funded academic library in the United States. It recently celebrated the acquisition of its nine millionth 
volume. The Library is a member of several consortia, many of which are dedicated to resource sharing. 
Physically the Library is made up of over 40 departmental units, each with a great deal of autonomy. The 
catalog of the University has been automated since the mid-1970s. In August of 1998 the interface was 
changed with a view to migrating to a web interface in the not-too-distant future. The database, however, 
has never been completely retro-converted, with the records for most items purchased prior to 1976 
accessible through author or title search only. Forty-five other academic libraries in the state combine with 
UIUC to create a union catalog called Illinet Online which provides statewide accessibility to over thirteen 
million items through patron initiated requests. The Illinois Library Computer System Office, or ILCSO, 
oversees the provision of this service statewide. The state has a model inter-library loan system that is 
multi-type in nature, reaching all levels from the school library to the special library, to the college and 
university library. The University is also a member of a consortium referred to as the Big Ten. This 
consortium, usually viewed as the basis for competitive sporting events, reaches beyond the arena and the 
court in the form of the Consortium for Institutional Cooperation or CIC. The CIC consists of the (eleven!) 
Big Ten Universities, the University of Chicago and the University of Illinois at Chicago. As the CIC name 
implies, cooperation extends into many areas including that of library resources. The result of the CIC's 
dedication to resource sharing is a Z39.50 based cooperative catalog called the CIC-VEL which also allows 
the patron to self-initiate requests for materials. The University also subscribes to the services of 
FirstSearch and in this manner provides access to the WorldCat bibliographic database, the web interface 
for users of the OCLC system. The result - patrons should be able to obtain whatever materials they wish, 
but they need to learn to work with three online catalogs. In addition, the University offers its users over 
seventy online databases, and uncounted CD-ROM databases. This embarrassment of riches combined 
with the decentralization of services frequently causes more problems than it solves for our users. 

The UIUC Library does not have a strong tradition in the dedication of resources to library use instruction. 
The decentralized nature of the system combined with the large numbers of incoming freshmen results in 
very few students being reached in a standard manner. Most instmction for the past five years had been 
delivered in the hit or miss manner of the course-integrated fifty minute session or generally voluntary 
attendance at a Library sponsored workshop. There was no budgetary commitment from the administration 
to library use instruction in most areas. The development of a Web-based tutorial, financed through grant 
funds, was meant to provide instruction to the 36,000 students enrolled at the University, at least until the 
new interface came on line. 

The University Librarian appointed a Task Force on User Education at the beginning of the 1997-98 
academic year. This task force was charged with the "design and implementation of steps for a plan to 
orient/instruct users of the University Library." [i] The plan was to involve both formal and informal 
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orientation and instruction in several media. Longer term consideration for training and orientation of all 
classes of users - undergraduate, graduate, faculty and staff were recommended for consideration. The 
University Librarian agreed to support the presentation of Web-based workshops on the use of article 
databases in the spring of 1998 with the hope that the skills learned there would be transferable to the new 
catalog interface. 

A large deficit for the Library had caused the cancellation of a large number of important science journals 
during the 1997/98 academic year. Swift and negative faculty reaction lead to the creation of a 
campus-wide "Task Force on the Future of the Library." After six weeks of intense study, several 
suggestions were made for the improvement of the Library's well-being, including an increased emphasis 
on user services. The Director of Collections and Assessment and the Head of Research and Planning for 
User Services joined to develop a survey to evaluate user attitudes toward the Library. This user survey 
disclosed some interesting findings. General attitudes toward the Library were, as expected, pretty positive. 
The area of greatest interest to the Task Force on User Education was the revelation that most individuals 
assessed themselves as very capable when it came to using the Library, giving themselves a "mean 
self-score of 3.72 of a possible 5" in their ability to find things in the library [2]. Even more amazing was 
the way in which they wished to learn - 45.5% on their own, 41.2% through web pages and online, and 
38.5% through individual instruction. [3] In the open ended questioning at the end of the survey, only 8% 
of respondents indicate they would like to see the library "Teach people to use the library, and advertise 
this service."[4] 

Jill Fatzer summed up the problem quite succinctly in her article on evaluation for the Library Literacy 
column in RQ over a decade ago. We need to look at the whole picture if we are going to do an effective 
job of evaluation. We need to look at the "affective, cognitive, and psychomotor areas." Fatzer further 
maintained that "these three domains closely parallel the traditional three aspects of competence: attitudes, 
knowledge, and skills." [5J In the recent past, the greatest emphasis in user surveys has been on evaluation, 
the first of these three domains, at the expense of the second two areas of competence. Perhaps the lack of 
tools such as those available through the World Wide Web, or perhaps a lack of enthusiasm for instruction, 
or perhaps a lack of confidence in our own knowledge has discouraged us. Whatever the reason, the time 
of inaction on our part must be put in the past. It is in fact imperative that we treat our students in the same 
manner as the biologist or engineer. We must dedicate ourselves to help them learn to be information 
literate, and unless we know the initial depth of knowledge of our users, we cannot assist with the learning 
process. With the development of World Wide Web tools that provide a great deal more interactivity, we 
should be able to easily and completely explore all three domains with our users. 

Users indicate a great deal of self-satisfaction with the manner in which they are accessing information at 
the library. They see little need for formal instruction, and frequently resent its imposition upon them. The 
question that occurs to us is this: Do they really know what they are doing? Can library users be 
successfully self-taught? Ultimately, we need to know what our students understand before we can 
structure a program to help them learn more about the library. Where do we begin? That is the question this 
pilot project on assessment seeks to answer. 

Methodology 

The first task was to determine what should be tested for. What are the competencies of the expert, or 
adequate library user needs to successfully navigate the sea of information we present to them? We are 
provided with a great deal of assistance here. The American Library Association has been developing the 
concept of information literacy through its National Information Literacy Initiative or NILI. Essentially, 
this program seeks to educate librarians in the delivery of information literacy skills, and to define the 
criteria by which the information literate may be judged. In addition, many colleges and universities 
define their own criteria for information literacy. There are some truly wonderful sites on the World Wide 
Web that can be used to develop a competency framework that suits the individual institution. [7] We have 
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based the assessment questions in our survey on a variety of the suggestions found in these resources as 
well as a general reading of the Big Six Skills articulated by Michael Eisenberg. [8] 

Our survey was divided into three distinct areas. First, users were asked to identify their relationship - 
frequently of use, type of use, etc. - with the library. They were also asked how they learned about the 
library and its use - self-taught, through a friend, through a reference desk worker, at a workshop, through 
the tutorial. The second part of the survey contained assessment questions which attempted to ascertain 
whether the individual knew how to articulate the problem, read a citation, the difference between an index 
and a catalog, information concerning primary, secondary, or tertiary sources. Boolean operators, keyword 
vs. subject searching, etc. Finally, some questions specific to the University's system were asked - what is 
the Gateway, can you search by subject in our catalog, where can I find the Social Sciences Abstracts, 
Dissertation Abstracts, or Poole's Index? All three sections will be treated as distinct parts of the study, but 
it is unlikely that the users will live up to their own expectations in general research, and even less likely in 
the navigation of the complex Library system at UIUC. 

The survey is initially being administered to a relatively small sample of students, all members of political 
science courses. This initial part of the project has required that we use paper surveys. If results indicate it 
should be continued, it will be mounted on the Web next year and extended across the education and social 
science disciplines of psychology, anthropology, political science, sociology, social work, speech 
communication and education. If this proves effective, it will be adjusted for audience type and extended 
across the disciplines to include the sciences and humanities. At this writing, results of the pilot survey are 
incomplete. From the results of the administration of a similar assessment tool to undergraduates at a small 
liberal arts college done from 1992-1995 however, it is unlikely that the general student population will 
score the 3.72 out of 5 that they had assumed they would. [£] 

Conclusion 

Our next step is fairly clear. The results of the Task Force on the Future of the Library and a changed 
budgetary process at the University required that strategic planning be done. One result of the new strategic 
plan for the library was the recommendation that the position of Instruction Coordinator be developed and 
funded in the near future. Along with this position comes support staff and a real budget for the 
implementation of a library-wide program. It is our sincere hope that the assessment we are doing now can 
be expanded across the curriculum and will help guide us to the areas in which instruction is most needed. 
Thus with the combined outreach of evaluation and assessment for our patrons we will both know what 
they want and need. Some areas of weakness can be addressed by information presented in workshops, 
some on the web, some might require the one-on-one venue of the reference desk. The delivery of 
instruction needs to be as varied as the types of learning styles. With increasing familiarity, the user might 
also feel more empowered to request changes that librarians have not considered, thus improving the 
services we render even further. User surveys alone are not enough, we need to continue and expand our 
attempts at user testing to provide a true picture of our surroundings. The user assessment and the user 
evaluation can, and must be combined. Schlichter and Pemberton do an excellent job of exploring the 
pitfalls of user studies as evaluative tools. FI 01 The addition of the user assessment will go a long way to 
eradicate the inherent problems in a strictly normative study. In the future, we might want to include the 
teaching faculty as another source of input concerning what truly is needed in the way of providing the best 
possible service to all of our publics. We then need to take our results into full account when doing the 
strategic planning we are all so involved in within our modern organizations. 
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